Skip to content

Conversation

Rentacookie
Copy link
Contributor

@Rentacookie Rentacookie commented Oct 23, 2024

Alpha support for MySQL as a replication datasource

Replication for MySQL is handled by listening for changes on the binlog which is similar to Postgres' write ahead log.

We use a forked version of the Zongji MySQL Binlog listener library that has been customised to our needs:
https://github.com/powersync-ja/powersync-mysql-zongji

Demo app and backend is available here:
https://github.com/powersync-ja/self-host-demo/blob/main/demos/nodejs-mysql/README.md

Some additional configuration is necessary for your instance of MySQL to enable the binlog. A MySQL user should also be created with the required permissions for Powersync to perform the replication.
https://github.com/powersync-ja/self-host-demo/tree/main/services/mysql/init-scripts

Some cleanup of modules package.json
Filter out tables not in sync rules
Improve abort logic handling
Replication Modules now implement this interface.
Removed singleton module exports
… to stop unwanted timezone skew

Added serverId configuration
Added check for binlog existence before starting replication.
@changeset-bot
Copy link

changeset-bot bot commented Oct 23, 2024

🦋 Changeset detected

Latest commit: 7c073b1

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 8 packages
Name Type
@powersync/service-core Minor
@powersync/service-module-mysql Minor
@powersync/service-sync-rules Minor
@powersync/service-module-mongodb Patch
@powersync/service-module-postgres Patch
@powersync/service-image Patch
test-client Patch
@powersync/service-jpgwire Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

stevensJourney
stevensJourney previously approved these changes Oct 23, 2024
Copy link
Collaborator

@stevensJourney stevensJourney left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good from my side. We might need to cleanup some of the TODO comments and track those items externally.

@Rentacookie Rentacookie marked this pull request as ready for review October 28, 2024 12:54
Copy link
Collaborator

@stevensJourney stevensJourney left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me

@Rentacookie Rentacookie merged commit e1f854f into feat/modular-replication-architecture Oct 28, 2024
9 checks passed
Copy link
Contributor

@rkistner rkistner left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Happy on the high-level structure, will see if I can still go through the details of the replication process at some point

# The service needs to be restarted for changes to take effect.

# Note that specifying the schema is currently required due to the default
# schema being specified as `public`, but in mysql the schema is the database name
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this still the case, with defaultSchema now being the database name?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No you are right, no longer needed :)

Comment on lines +48 to +50
// await connection.query(`DROP DATABASE IF EXISTS ${TEST_CONNECTION_OPTIONS.database}`);
//
// await connection.query(`CREATE DATABASE IF NOT EXISTS ${TEST_CONNECTION_OPTIONS.database}`);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can these comments be removed?

"@powersync/service-core": "workspace:*",
"@powersync/service-sync-rules": "workspace:*",
"@powersync/service-types": "workspace:*",
"@powersync/mysql-zongji": "0.0.0-dev-20241023144335",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be good to get a proper release out for this.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Definitely. I'm still struggling with an issue where the testing framework of that zongji lib causes the build to fail even though all the tests pass. That is currently the main blocker for getting out a proper released version.

@rkistner rkistner deleted the feat/mysql-alpha branch December 9, 2024 07:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants